Spatially and temporally coherent multi-LiDAR point cloud fusion

ABSTRACT

A scanning system includes a plurality of light detection and ranging (LiDAR) sensors, each LiDAR sensor of the plurality of LiDAR sensors being configured to generate a point cloud based on interactions of emitted light with a surrounding environment, and a processor configured to split a full 360° rotation of each LiDAR sensor into a plurality of slices including a first slice and a second slice, to generate a coherent fused point cloud by fusing together portions of point clouds of the plurality of LiDAR sensors corresponding to the first slice, and fusing together portions of point clouds of the plurality of LiDAR sensors corresponding to the second slice, to determine an action for the scanning system to take in response to the generation of the coherent fused point cloud, and to cause the scanning system to implement the action.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application claims priority to, and the benefit of, U.S.Provisional Application No. 62/960,618 (“SPATIALLY AND TEMPORALLYCOHERENT MULTI-LIDAR POINT CLOUD FUSION”), filed on Jan. 13, 2020, theentire content of which is incorporated herein by reference.

FIELD

Aspects of embodiments of the present disclosure are generally relatedto three-dimensional perception using point cloud data.

BACKGROUND

Generally, technical areas such as autonomous driving, robotics, andadvanced driver-assistance (ADAS) systems rely on three-dimensionalperception capabilities. The understanding of where an object is locatedin the world with respect to the robotic or autonomous driving platformis crucial for the decision making. Moreover, such information may onlybe valuable if one can associate an accurate timestamp to thelocalization of an object. Given a state of the world and a time, theprediction module can determine how agents in a scene are behaving, thusallowing the decision-making module to maneuver safely (or, e.g., informthe driver of potential hazards) in the environment.

The above information disclosed in this Background section is only forenhancement of understanding of the present disclosure, and therefore itmay contain information that does not form the prior art that is alreadyknown to a person of ordinary skill in the art.

SUMMARY

Aspects of embodiments of the present disclosure are directed to asystem and method for assuring spatio-temporal consistency betweensensors of a multi-LiDAR system to achieve accurate fused multi-LiDARpoint clouds.

According to some embodiments of the present disclosure, there isprovided a scanning system including: a plurality of light detection andranging (LiDAR) sensors, each LiDAR sensor of the plurality of LiDARsensors being configured to generate a point cloud based on interactionsof emitted light with a surrounding environment; and a processorconfigured to split a full 360° rotation of each LiDAR sensor into aplurality of slices including a first slice and a second slice, togenerate a coherent fused point cloud by fusing together portions ofpoint clouds of the plurality of LiDAR sensors corresponding to thefirst slice, and fusing together portions of point clouds of theplurality of LiDAR sensors corresponding to the second slice, todetermine an action for the scanning system to take in response to thegeneration of the coherent fused point cloud, and to cause the scanningsystem to implement the action.

In some embodiments, the processor is configured to generate thecoherent fused point cloud by: assigning a first timestamp to theportions of the point clouds of the plurality of LiDAR sensorscorresponding to the first slice; and assigning a second timestamp tothe portions of the point clouds of the plurality of LiDAR sensorscorresponding to the second slice.

In some embodiments, separately fusing together portions of the pointclouds of the plurality of LiDAR sensors corresponding to the firstslice and the second slice includes: fusing together ones of the pointclouds associated with the first timestamp; and separately fusingtogether ones of the point clouds associated with the second timestamp.

In some embodiments, the first slice of each LiDAR sensor is earlier inrotation that the second slice, and wherein the processor is configuredto process the portions of the point clouds corresponding to the firstslice, while the plurality of LiDAR sensors are capturing the portionsof the point clouds corresponding to the second slice.

In some embodiments, the scanning system further includes: a controllerconfigured to control trigger times of rotations of the plurality ofLiDAR sensors, and to phase lock first ones of the plurality of LiDARsensors and to phase lock second ones of the plurality of LiDAR sensors.

In some embodiments, the controller is configured to phase lock thefirst ones of the plurality of LiDAR sensors at a substantially samephase angle and to phase lock the second ones of the plurality of LiDARsensors at a substantially same phase angle.

In some embodiments, at any given time, the first ones of the pluralityof LiDAR sensors face in a first direction and the second ones of theplurality of LiDAR sensors face a second direction substantiallyopposite the first direction.

In some embodiments, at any given time, the first and second ones of theplurality of LiDAR sensors face in a substantially same direction.

In some embodiments, the plurality of LiDAR sensors are positioned at atleast two corners of the scanning system.

In some embodiments, the scanning system includes a vehicle, and thefirst and second slices respectively correspond to a front side and arear side of the vehicle.

In some embodiments, the scanning system includes a vehicle, and thefirst and second slices respectively correspond to a left side and aright side of the vehicle.

According to some embodiments of the present disclosure, there isprovided a scanning system including: a platform; a plurality of lightdetection and ranging (LiDAR) sensors mounted on the platform, eachLiDAR sensor of the plurality of LiDAR sensors being configured togenerate a point cloud based on interactions of emitted light with asurrounding environment; a controller configured to control triggertimes of rotations of the plurality of LiDAR sensors; and a processorconfigured to split a full 360° rotation of each LiDAR sensor into aplurality of slices including a first slice and a second slice, and togenerate a coherent fused point cloud by fusing together portions ofpoint clouds of the plurality of LiDAR sensors corresponding to thefirst slice, and fusing together portions of point clouds of theplurality of LiDAR sensors corresponding to the second slice.

In some embodiments, the processor is configured to generate thecoherent fused point cloud by: assigning a first timestamp to theportions of the point clouds of the plurality of LiDAR sensorscorresponding to the first slice; and assigning a second timestamp tothe portions of the point clouds of the plurality of LiDAR sensorscorresponding to the second slice.

In some embodiments, separately fusing together portions of the pointclouds of the plurality of LiDAR sensors corresponding to the firstslice and the second slice includes: fusing together ones of the pointclouds associated with the first timestamp; and separately fusingtogether ones of the point clouds associated with the second timestamp.

In some embodiments, the first slice of each LiDAR sensor is earlier inrotation that the second slice, and wherein the processor is configuredto process the portions of the point clouds corresponding to the firstslice, while the plurality of LiDAR sensors are capturing the portionsof the point clouds corresponding to the second slice.

In some embodiments, the controller is further configured to phase lockfirst ones of the plurality of LiDAR sensors at a substantially samephase angle and to phase lock second ones of the plurality of LiDARsensors at a substantially same phase angle.

In some embodiments, at any given time, the first ones of the pluralityof LiDAR sensors face in a first direction and the second ones of theplurality of LiDAR sensors face a second direction substantiallyopposite the first direction.

In some embodiments, at any given time, the first and second ones of theplurality of LiDAR sensors face in a substantially same direction.

According to some embodiments of the present disclosure, there isprovided a method of generating a coherent fused point cloud by ascanning system including a plurality of light detection and ranging(LiDAR) sensors, the method including: splitting a full 360° rotation ofeach LiDAR sensor of the plurality of LiDAR sensors into a plurality ofslices including a first slice and a second slice, each LiDAR sensorbeing configured to generate a point cloud based on interactions ofemitted light with a surrounding environment; and generating a coherentfused point cloud by separately fusing together portions of the pointclouds of the plurality of LiDAR sensors corresponding to the firstslice and the second slice.

In some embodiments, the method further includes: phase locking firstones of the plurality of LiDAR sensors at a substantially same phaseangle and phase locking second ones of the plurality of LiDAR sensors ata substantially same phase angle, wherein the first and second ones ofthe plurality of LiDAR sensors face in a substantially same direction orin substantially opposite directions.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawings will be provided by the Office upon request and paymentof the necessary fee.

The accompanying drawings, together with the specification, illustrateexample embodiments of the present disclosure, and, together with thedescription, serve to explain the principles of the present disclosure.

FIGS. 1A and 1B respectively illustrate a scanning system including twoLiDAR sensors that are phase locked at 90 degrees at a first rotationtime and a second rotation time.

FIGS. 2A and 2B respectively illustrate a scanning system including twoLiDAR sensors that are phase locked at zero degrees at a first rotationtime and a second rotation time.

FIG. 3 illustrates a related-art LiDAR scanning of an object withrelative motion between the object and a multi-LiDAR vehicle resultingin spatio-temporal inconsistencies between the different LiDARs of themulti-LiDAR vehicle.

FIG. 4A illustrates an example of the related art in which fused pointclouds are obtained from 5 different LiDAR sensors for a given scene,and the sensed car boundaries exhibit shadowing effects.

FIG. 4B illustrates a comparative example in which a point cloud isobtained from a single LiDAR sensor for the same scene, and the sensedcar boundaries do not exhibit any shadowing effect.

FIG. 5 illustrates an artifact at the cut angle boundary, which is atthe rear of a vehicle and parallel to the vehicle motion.

FIG. 6 illustrates an artifact at the cut angle boundary, which isperpendicular to vehicle motion and to the left.

FIG. 7A illustrates a scanning system for generating a coherent fusedLiDAR point cloud, according to some embodiments of the presentdisclosure.

FIG. 7B illustrates a first phase of a single-phase angle LiDAR splitduring which all LiDAR sensors are facing in a same direction, accordingto some embodiments of the present disclosure.

FIG. 7C illustrates a second phase of the single-phase angle LiDAR splitduring which all LiDAR sensors are facing in the opposite direction,according to some embodiments of the present disclosure.

FIGS. 8A and 8B illustrate the captured cloud point during the firstphase and the second phase, respectively, of the LiDAR rotation,according to some embodiments of the present disclosure.

FIG. 9A illustrates a first phase of a dual-phase angle LiDAR split inwhich a first set of LiDAR sensors are facing in a first direction and asecond set of LiDAR sensors are facing in a second direction oppositethe first direction, according to some embodiments of the presentdisclosure.

FIG. 9B illustrates a second phase of the dual-phase angle LiDAR splitduring which the first set of LiDAR sensors are facing in seconddirection and the second set of LiDAR sensors are facing in the firstdirection, according to some embodiments of the present disclosure.

FIG. 10A illustrates the point cloud from the front and top sensorsduring the first phase of the rotation split, according to someembodiments of the present disclosure.

FIG. 10B illustrates the point cloud from the alternative top down view,according to some embodiments of the present disclosure.

FIG. 10C illustrates the point cloud from the rear sensors during thefirst phase of the rotation split, according to some embodiments of thepresent disclosure.

DETAILED DESCRIPTION

The detailed description set forth below is intended as a description ofexample embodiments of a system and method for generating a coherentfused point cloud in a multi-LiDAR system, provided in accordance withthe present disclosure, and is not intended to represent the only formsin which the present disclosure may be constructed or utilized. Thedescription sets forth the features of the present disclosure inconnection with the illustrated embodiments. It is to be understood,however, that the same or equivalent functions and structures may beaccomplished by different embodiments that are also intended to beencompassed within the scope of the disclosure. As denoted elsewhereherein, like element numbers are intended to indicate like elements orfeatures.

To permit a three-dimensional understanding of the environment,autonomous platforms may be equipped with multiple sensors. Sincespatial and temporal accuracy is desired to act, any inconsistenciesresulting from sensor disagreements can have a catastrophic impact onthe vehicle's decisions.

A light detection and ranging (LiDAR) device is a time-of-flightlaser-based scanning device that emits multiple beams at high frequencyand concurrently receives returns as beams hit objects. LiDAR may returnbetween 200,000 and 1,000,000 points per second. The collection ofpoints obtained from laser beam returns bouncing on surrounding objectsmay be referred to as a point cloud. One commonly used type of LiDAR isa spinning device that generates a 360-degrees point cloud of thesurroundings of the vehicle at 10 Hz to 20 Hz. In order to get acomplete coverage of the environment close to the platform and atgreater distance, multiple sensors may be mounted onto a singleplatform. A great challenge is to merge data received by multiplesensors, which may be of different type and placed at differentlocations on the platform, both in time and in space. This may bereferred to as sensor fusion. Naively starting the sensors and capturinga 360 degrees output will lead to an inconsistent fused point cloud, asvehicles hit by the laser beams may be seen at different times and indifferent locations by the various LiDAR sensors. To compensate forthese shortcomings, it is desirable to address two issues: the spatialstatic consistency and the dynamic spatio-temporal consistency of theLidar points.

The static spatial consistency can be addressed through calibration ofthe sensors, that is, by determining precisely the location of thesensors relative to each other. In a perfectly static scene, propercalibration of sensors will allow the points captured by each LiDAR tobe transformed into a common reference frame in order to create acoherent fused point cloud.

However, as soon as objects are dynamic and have relative motion withregard to the platform, which may be the case in robotics applications,the various LiDAR sources don't receive returns off the objects at thesame time. This issue may be referred to as dynamic spatio-temporalinconsistency. The LiDARs may capture information while spinning over aperiod of 50 ms to 100 ms (i.e., 20 Hz or 10 Hz). During that timelapse, objects might have moved by a few meters depending on theirrelative speed. Indeed, a vehicle driving in the opposite direction oftraffic with respect to the sensor platform at an urban speed of 25 mph(relative speed between the two vehicles may be 50 mph) may have movedrelative to the sensor 2.24 meters in 100 ms. At highway speeds, therelative position of the vehicle with respect to the sensor may, forexample, change by 5.8 meters in the same amount of time. Hence if afirst sensor sees an object at the beginning of its rotation (spin) andsecond sensor sees it at the end of the rotation and the two pointclouds are fused together, one will get a smeared effect or a ghostingeffect: a car bumper for instance will appear twice at differentlocations in the fused point cloud.

Thus, some embodiments of the present disclosure are directed to asystem and method for resolving or improving the spatio-temporalinconsistency in sensor fusion of robotic platforms to achieve accuratefused multi-LiDAR point clouds. According to some embodiments, thesystem is capable of providing a coherent spatial and temporal fusedpoint cloud output by phase-locking light detection and ranging (LiDAR)sensors at precisely calibrated cut angles, that is, the position atwhich the revolution (spin) for the point cloud capture is initiated,and by splitting the LiDAR point clouds in time to prevent points in aregion captured at overly different times to be associated with the sametimestamp. Accordingly, the system increases the point cloud outputfrequency, and reduces the latency in the pipeline by processing LiDARpoints before the end of a rotation. Herein, a point cloud is acollection of points that represent a 3D shape or feature. Each pointhas its own set of X, Y and Z coordinates and, in some examples, has anassociated timestamp recording time of capture.

Hereinafter, the cut angle of a spinning LiDAR refers to the directionat which the capture of each new point cloud starts. In some examples,the LiDAR may capture a full 360 degrees point cloud from cut angle overa period of 50 ms, when the device rotates at 20 Hz, or a period of 100ms, when it rotates at 10 Hz. Further, phase locking two LiDARs refersto setting (and fixing) the cut angle of one sensor at a relative anglewith respect to the cut angle of the other sensor (e.g. when the twoLiDARs have same or similar rotation frequencies, such as frequencieswithin 5 Hz or less of each other, within 2 Hz or less of each other,within 1 Hz or less of each other, or within 0.5 Hz or less of eachother). Thus, once two sensors are phase locked, the direction in whicha LiDAR sensor points to can be determined, at any point in time, byknowing the direction in which the other LiDAR sensor points to at thattime. When, the relative phase is set to 0 degrees, the two LiDARsensors have the same angle.

FIGS. 1A and 1B respectively illustrate a scanning system 10 includingtwo LiDAR sensors 100 that are phase locked at 90 degrees at a firstrotation time t1 and a second rotation time t2. FIGS. 2A and 2Brespectively illustrate a scanning system 10 including two LiDAR sensors100 that are phase locked at zero degrees at a first rotation time t1and a second rotation time t2.

As each spinning LiDAR sensor 100 provides small packets of points, thescanning system 10, can collect a full rotation of points and consumethe full 360 degrees point cloud, or the the scanning system 10 can,according to some embodiments, collect slices of the 360 degrees andtimestamp these intermediate slices individually. The creation ofintermediate slices is referred to herein as the splitting of the LiDARpoint cloud. Performing one slice/split results in outputting twohemispheres of points, and performing two slices results in three thirdsof a sphere of points, in some embodiments in which the slices are ofequal or similar size. In other embodiments, the slices are not of equalor similar size.

FIG. 3 illustrates a related-art LiDAR scanning of an object 11 withrelative motion between the object 11 and a multi-LiDAR vehicle 12resulting in spatio-temporal inconsistencies between the differentLiDARs of the multi-LiDAR vehicle.

Referring to FIG. 3 , a multi-LiDAR vehicle 12 of the related art may beequipped with multiple sensors (e.g., two LiDAR sensors) 100 that arenot phase locked, or where a phase lock between the multiple sensors isnot appropriately tuned and/or where the phase difference is notappropriately compensated for in processing of the sensor data. In adynamic scene, such as when there is relative motion between an object11 (e.g., an incoming vehicle) and the platform, because the LiDARsensors 100 can be pointing at different locations at different times,they may observe a region or the object at different times. As a result,instead of obtaining the same crisp boundaries of an object as may bethe case with a single LiDAR sensor 100, the object 11 may appear ashaving a shadow of points.

FIG. 4A illustrates an example of the related art in which fused pointclouds are obtained from 5 different LiDAR sensors for a given scene,and the sensed car boundaries exhibit shadowing effects. FIG. 4Billustrates a comparative example in which a point cloud is obtainedfrom a single LiDAR sensor for the same scene, and the sensed carboundaries do not exhibit any shadowing effect. In FIG. 4A, each colorrepresents a different LiDAR sensor.

As illustrated in FIG. 4A, the shadowing effect (e.g., the 3-4boundaries of the back car shown in regions A and B of FIG. 4A) may makeit difficult to precisely determine a location of an object given atimestamp, at a particular point. The uncertainty resulting from theshadowing effect may propagate into an algorithm that uses it as well asthe annotation of the data in the case of supervised algorithms.Furthermore, the location of the LiDAR sensors on the multi-LiDARvehicle may influence the artifact shapes that are generated. As aresult, it may be difficult to train an algorithm on one dataset andthen to generalize to another dataset, because the uncertainty maydepend on the setup.

Another noticeable artifact that may appear when doing fusion of datafrom non-phase locked sensors is related to discontinuities at theboundary of a split. Because points on one side of the split boundaryhave been collected at the beginning of the LiDAR rotation and points atthe other side of the split boundary have been collected at the end ofthe rotation, discontinuous object shapes may be observed when there isrelative motion between the LiDAR sensor and the object.

FIG. 5 illustrates an artifact 24 at the cut angle boundary, which is atthe rear of a vehicle and parallel to the vehicle motion. FIG. 6illustrates an artifact at the cut angle boundary, which isperpendicular to vehicle motion and to the left. In FIG. 6 , each colorrepresents a different LiDAR sensor 100.

In the example of FIG. 5 , the discontinuity artifact 24 appears in therear area of the multi-LiDAR vehicle 12 and an approaching car is splitalong the X direction. In other examples, such as the example of FIG. 6, when the cut angle is set perpendicular to the motion of themulti-LiDAR vehicle 12 (e.g., set to the side of the multi-LiDAR vehicle12, e.g., along the Y axis), the artifacts may be less predictable asthe observed shapes of surrounding objects are more impacted by themotion of the multi-LiDAR vehicle, and can result in more than a sliding(or perceived discontinuity) of half of the object. The artifactsobserved in this case correspond to a similar shadowing effect but inthe width direction of the cars. Hence, the pickup truck may appear tobe 50 cm or 1 meter wider than it actually is. Similarly, the car to theleft of the pickup truck may appear deformed. Inaccurate measurementslike these could prevent the multi-LiDAR vehicle 12 from, for example,finding a parking spot next to that car.

FIG. 7A illustrates a scanning system 10 for generating a coherent fusedLiDAR point cloud, according to some embodiments of the presentdisclosure.

According to some embodiments, the scanning system 10 includes aplatform (e.g., a vehicle or a robotic platform) 20, a plurality ofLiDAR sensors 100 positioned at various points on or in the platform 20,a controller 200 (also referred to as a synchronization module orcontrol circuit) configured to control the motion (e.g., rotation) andtrigger times of the LiDAR sensors 100, and a processor (e.g., aprocessor circuit) 300 for processing the data received from each LiDARsensor, which may represent at least a slice of the full 360 degreespoint cloud. As used herein, references to the motion or rotation of aLiDAR sensor can refer to physical movement of the sensor itself (or oneor more components thereof), or can refer to movement or rotation of avolume, area, or region sensed by the LiDAR sensor. The controller 200operates each LiDAR sensor 100 such that a 360 degrees rotation of theLiDAR sensor 100 is split into a plurality of slices. Because each slicerepresents a shorter time interval than a full 360 degrees rotation, arotation slice effectively reduces the spatio-temporal inconsistencydescribed above. Thus, the scanning system 10 can eliminate artifactscaused by drastically different time captures in areas around theplatform 20, for instance at the cut angle locations. According to someembodiments, the processor 300 is configured to split a full 360°rotation of each LiDAR sensor 100 into a plurality of slices including afirst slice and a second slice, and to generate a coherent fused pointcloud by separately fusing together portions of point clouds of theLiDAR sensors 100 corresponding to the first slice and the second slice.In so doing, the processor 300 may assign a first timestamp to theportions of the point clouds of the LiDAR sensors 100 that correspond tothe first slice, and assign a second timestamp to the portions of thepoint clouds corresponding to the second slice. The processor may thenfuse together ones of the point clouds associated with the firsttimestamp, and separately fuse together ones of the point cloudsassociated with the second timestamp. This allows the processor 300 orany downstream algorithm to process the portions of the point cloudscorresponding to the first slice, while the LiDAR sensors 100 arecapturing the portions of the point clouds corresponding to the secondslice, which improves pipelining.

The point cloud may then be used by various modules of the perceptionpipeline. One such module may be the clustering algorithm. Given aninput point cloud covering part of the environment around the scanningsystem 10, the clustering algorithm may identify point clusterscorresponding to other agents (vehicles, pedestrians, obstacles) anddetermine their shape, location, and orientation. Another such modulemay be a deep-learning-based 3D object detection algorithm that may tryto obtain similar output from the point cloud with the additionalcapability of determining the type of object looked at. The clusters maythen be passed on to a tracking algorithm that may try to add temporalinformation to these measurements, as the system may have a sense ofwhere the object currently is but want to understand how it's moving bycombining that information with prior measurements of the same object.This may then be used to predict an agent's trajectory and actaccordingly.

A different scenario may be to use the coherent points gathered toestimate the road surface (e.g., shape or inconsistencies on theground). By having coherent lidar points, one has better chances atdetermining the manifold on which the points lie. The road surfaceinformation may then be used by the vehicle controller to, for example,slow down ahead of a speed bump or accelerate more when on a hill thanwhen on a flat road.

In some embodiments, the controller 200 phase locks ones of the LiDARsensors 100 that cover a common slice of the rotation at a same phaseangle (e.g., a zero-degree angle) to allow for spatio-temporalconsistency. Given that the locations of the LiDAR sensors 100 on thescanning system 10 may be different, having the same phase angle doesnot necessarily guarantee that each LiDAR observes every regionsimultaneously, but reduces the time lapse between two LiDAR captures.In some embodiments, LiDAR sensors that do not cover the same slice maynot be phase locked at the same angle (e.g., may be phase locked atdifferent angles). The controller 200 is configured to trigger therotation of all LiDAR sensors 100 substantially simultaneously for allof the slices. The processor 300 fuses point clouds originating fromvarious LiDAR sensors over a single slice/phase of the rotation forfurther processing.

Slicing the 360 degrees rotation and phase-locking sensors covering thesame slice at zero degrees, according to some embodiments, not onlyaddress both types of artifacts described above (the shadowing anddiscontinuity artifacts) but also provide the following additionalbenefits: 1) they reduce the latency in the pipeline as the processor300 can process a slice of the point cloud rather than waiting for thefull 360 degree rotation to be completed, which can be valuable inreal-time applications, such as autonomous driving, 2) they allow forsmaller point clouds to be processed by the processor and/or any otherdownstream algorithm, and 3) allow for more accurate timestamping of thecaptured cloud points. In effect, rotational slicing and phase-lockingsensors covering the same slice smooths out the workload of theprocessor and reduces the bandwidth taken by the LiDAR sensors 100.

FIG. 7B illustrates a first phase of a single-phase angle LiDAR splitduring which all LiDAR sensors 100 are facing in a same direction,according to some embodiments of the present disclosure. FIG. 7Cillustrates a second phase of the single-phase angle LiDAR split duringwhich all LiDAR sensors 100 are facing in the opposite direction,according to some embodiments of the present disclosure. FIGS. 8A and 8Billustrate the captured cloud point during the first phase and thesecond phase, respectively, of the LiDAR rotation, according to someembodiments of the present disclosure. In FIGS. 8A-8B, each colorrepresents a different LiDAR sensor.

Referring to FIGS. 7B-7C, in some embodiments, the scanning system 10(e.g., an ego vehicle) includes 5 LiDAR sensors 100 and a controller 200allowing for simultaneous triggering of the LiDAR rotations. Accordingto some examples, a single 32-line LiDAR sensor 100-3 may be mounted ontop of the platform 20 and four 16-lines LiDAR sensors 100-1, 100-2,100-4, and 100-5 may be mounted in all 4 corners of the platform 20. Insome embodiments, the system performs a single split per rotation (e.g.,per 360 degrees/full rotation), which allows for consistent inter-lidarinformation without cutting too many objects of interests around thescanning system 10. However, embodiments of the present disclosure arenot limited thereto, and the rotation may be cut/split into any suitablenumber of slices. Given that the areas of interest may be located infront or back of the platform 20, in some embodiments, the LiDAR pointcloud may be split into a front slice and a rear slice. In some example,the LiDAR point cloud may also be cut in a left slice and a right sliceof the environment, however, this may lead to more objects in the frontand in the back of the platform 20 being cut in two. Splitting objectsmay make it harder for a 3D detection algorithm to consume the data anddetect the objects at the boundary of the split. Given that it is morecommon to have cars driving in the same direction as the scanning system10, the processor 300 may cut a higher number of vehicles when splittingthe point cloud in the x/−x direction (e.g., into left and right splits)than when splitting the point cloud in the y or −y direction (e.g., intofront and rear splits).

In the examples of FIGS. 7B and 7C, in which all LiDAR sensors 100 ofthe system 10 scan the same slice, all LiDARs face forward during afirst phase (FIG. 7A) and face backward during a second phase. Accordingto some embodiments, the half point clouds obtained by the scanningsystem 10 do not exhibit any shadowing or spatial discontinuityartifacts. That is, no shadowing effect may be observed on the objectsin the scene whether they are static or dynamic, and no artifact may beobserved at the cut angles (e.g., 90-degree or 270-degree angle) sinceno drastically temporally different points have been obtained from thevarious LiDAR sensors 100. In some embodiments, the result is aspatio-temporally consistent fused multi-LiDAR point cloud shown inFIGS. 8A and 8B. In FIGS. 8A-8B, which illustrates accurate vehicleboundaries with no shadowing effect, different colors correspond todifferent lidar sensors.

According to some embodiments, during each phase of the rotation, someof the LiDAR sensors 100 may be pointing in one direction, while othersensors may be pointing at the opposite phase angles.

As used herein, the when two or more sensors “face in the substantiallysame direction”, are “phase locked at substantially the same phaseangle”, or are “phase locked at zero degree angles”, the phase angles ofthe sensors may not be exactly identical, but may be within a narrowrange of one another to account for timing constraints, calibrationconstraints, and/or the like. In some examples, the range may be 5degrees, 2 degrees, or less. Similarly, when two sensors face insubstantially opposite directions, their phases may be within a range of180 degrees, which may account for timing constraints, calibrationconstraints, and/or the like. In some examples, the range may be 5degrees, 2 degrees, or less.

FIG. 9A illustrates a first phase of a dual-phase angle LiDAR split inwhich a first set of LiDAR sensors 100 are facing in a first direction(e.g., the x direction) and a second set of LiDAR sensors 100 are facingin a second direction opposite the first direction (e.g., the −xdirection), according to some embodiments of the present disclosure.FIG. 9B illustrates a second phase of the dual-phase angle LiDAR splitduring which the first set of LiDAR sensors 100 are facing in seconddirection and the second set of LiDAR sensors 100 are facing in thefirst direction, according to some embodiments of the presentdisclosure.

Referring to FIGS. 9A and 9B, according to some examples, three of thefive LiDAR sensors (e.g., the two forward LiDAR sensors 100-1 and 100-2and the top LiDAR sensor 100-3) are phase-locked to face in the forwarddirection and the remaining two sensors (e.g., the rear LiDAR sensors100-4 and 100-5) are phase-locked to face backward during a first phaseduring a first rotational phase (FIG. 9A), and the three LiDAR sensorsmay face backward and the two rear LiDAR sensors face forward during asecond rotational phase (FIG. 9B). A feature of this configuration isthat each phase entails information about a large part of the 360degrees field of view, while the point clouds obtained in thisconfiguration offer spatio-temporal consistency with limited (e.g.,minimal areas) subject to dynamic artifacts.

FIG. 10A illustrates the point cloud from the front and top sensorsduring the first phase of the rotation split, according to someembodiments of the present disclosure; FIG. 10B illustrates the pointcloud from the alternative top down view, according to some embodimentsof the present disclosure; and FIG. 10C illustrates the point cloud fromthe rear sensors during the first phase of the rotation split, accordingto some embodiments of the present disclosure. In FIGS. 10A-10B, eachcolor represents a different LiDAR sensor.

As described herein, the scanning system according to some embodimentsprovides a solution for spatio-temporal consistent multi-LiDAR pointcloud. To mitigate temporal inconsistency between LiDARs, the scanningsystem according to some embodiments splits the 360 degrees point cloudat runtime and locks LiDAR sensors that are scanning the same part ofthe environment at the same phase angle. This prevents or substantiallyreduces instances of artifacts, such as shadowing of objects andinconsistent information in the regions of the cut angles, and alsoreduces the latency in an online processing pipeline as the data for thefirst slice can be processed as soon as that slice has been scanned andnot at the end of the LiDAR rotation. Furthermore, the system allows forhigher temporal accuracy of when the point cloud has been captured.

It will be understood that, although the terms “first”, “second”,“third”, etc., may be used herein to describe various elements,components, regions, layers, and/or sections, these elements,components, regions, layers, and/or sections should not be limited bythese terms. These terms are used to distinguish one element, component,region, layer, or section from another element, component, region,layer, or section. Thus, a first element, component, region, layer, orsection discussed below could be termed a second element, component,region, layer, or section, without departing from the scope of theinventive concept.

Spatially relative terms, such as “beneath”, “below”, “lower”, “under”,“above”, “upper” and the like, may be used herein for ease ofdescription to describe one element or feature's relationship to anotherelement(s) or feature(s) as illustrated in the figures. It will beunderstood that the spatially relative terms are intended to encompassdifferent orientations of the device in use or in operation, in additionto the orientation depicted in the figures. For example, if the devicein the figures is turned over, elements described as “below” or“beneath” or “under” other elements or features would then be oriented“above” the other elements or features. Thus, the example terms “below”and “under” can encompass both an orientation of above and below. Thedevice may be otherwise oriented (e.g., rotated 90 degrees or at otherorientations) and the spatially relative descriptors used herein shouldbe interpreted accordingly. In addition, it will also be understood thatwhen a layer is referred to as being “between” two layers, it can be theonly layer between the two layers, or one or more intervening layers mayalso be present.

The terminology used herein is for the purpose of describing particularembodiments and is not intended to be limiting of the inventive concept.As used herein, the singular forms “a” and “an” are intended to includethe plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “include”,“including”, “comprises”, and/or “comprising”, when used in thisspecification, specify the presence of stated features, integers, steps,operations, elements, and/or components, but do not preclude thepresence or addition of one or more other features, integers, steps,operations, elements, components, and/or groups thereof. As used herein,the term “and/or” includes any and all combinations of one or more ofthe associated listed items. Expressions such as “at least one of”, whenpreceding a list of elements, modify the entire list of elements and donot modify the individual elements of the list. Further, the use of“may” when describing embodiments of the inventive concept refers to“one or more embodiments of the inventive concept”. Also, the term“exemplary” is intended to refer to an example or illustration.

It will be understood that when an element or layer is referred to asbeing “on”, “connected to”, “coupled to”, or “adjacent” another elementor layer, it can be directly on, connected to, coupled to, or adjacentthe other element or layer, or one or more intervening elements orlayers may be present. When an element or layer is referred to as being“directly on”, “directly connected to”, “directly coupled to”, or“immediately adjacent” another element or layer, there are nointervening elements or layers present.

As used herein, the terms “use”, “using”, and “used” may be consideredsynonymous with the terms “utilize”, “utilizing”, and “utilized”,respectively.

The scanning system and/or any other relevant devices or componentsaccording to embodiments of the present disclosure described herein,such as the controller and processor, may be implemented by utilizingany suitable hardware, firmware (e.g., an application-specificintegrated circuit), software, or any suitable combination of software,firmware, and hardware. For example, the various components of thescanning system may be formed on one integrated circuit (IC) chip or onseparate IC chips. Further, the various components of the scanningsystem may be implemented on a flexible printed circuit film, a tapecarrier package (TCP), a printed circuit board (PCB), or formed on thesame substrate. Further, the various components of the scanning systemmay be a process or thread, running on one or more processors, in one ormore computing devices, executing computer program instructions andinteracting with other system components for performing the variousfunctionalities described herein. The computer program instructions arestored in a memory which may be implemented in a computing device usinga standard memory device, such as, for example, a random access memory(RAM). The computer program instructions may also be stored in othernon-transitory computer-readable media such as, for example, a CD-ROM,flash drive, or the like. Also, a person of skill in the art shouldrecognize that the functionality of various computing devices may becombined or integrated into a single computing device, or thefunctionality of a particular computing device may be distributed acrossone or more other computing devices without departing from the scope ofthe exemplary embodiments of the present disclosure.

While this disclosure has been described in detail with particularreferences to illustrative embodiments thereof, the embodimentsdescribed herein are not intended to be exhaustive or to limit the scopeof the disclosure to the exact forms disclosed. Persons skilled in theart and technology to which this disclosure pertains will appreciatethat alterations and changes in the described structures and methods ofassembly and operation can be practiced without meaningfully departingfrom the principles, and scope of this disclosure, as set forth in thefollowing claims and equivalents thereof.

What is claimed is:
 1. A scanning system comprising: a plurality oflight detection and ranging (LiDAR) sensors, each LiDAR sensor of theplurality of LiDAR sensors being configured to generate a point cloudbased on interactions of emitted light with a surrounding environment; aprocessor configured to split a full 360° rotation of each LiDAR sensorinto a plurality of slices comprising a first slice and a second slice,to generate a coherent fused point cloud by fusing together portions ofpoint clouds of the plurality of LiDAR sensors corresponding to thefirst slice, and fusing together portions of point clouds of theplurality of LiDAR sensors corresponding to the second slice, todetermine an action for the scanning system to take in response to thegeneration of the coherent fused point cloud, and to cause the scanningsystem to implement the action; and a controller configured to controltrigger times of rotations of the plurality of LiDAR sensors, and tophase lock first ones of the plurality of LiDAR sensors and to phaselock second ones of the plurality of LiDAR sensors.
 2. The scanningsystem of claim 1, wherein the processor is configured to generate thecoherent fused point cloud by: assigning a first timestamp to theportions of the point clouds of the plurality of LiDAR sensorscorresponding to the first slice; and assigning a second timestamp tothe portions of the point clouds of the plurality of LiDAR sensorscorresponding to the second slice.
 3. The scanning system of claim 2,wherein separately fusing together portions of the point clouds of theplurality of LiDAR sensors corresponding to the first slice and thesecond slice comprises: fusing together ones of the point cloudsassociated with the first timestamp; and separately fusing together onesof the point clouds associated with the second timestamp.
 4. Thescanning system of claim 1, wherein the first slice of each LiDAR sensoris earlier in rotation that the second slice, and wherein the processoris configured to process the portions of the point clouds correspondingto the first slice, while the plurality of LiDAR sensors are capturingthe portions of the point clouds corresponding to the second slice. 5.The scanning system of claim 1, wherein the controller is configured tophase lock the first ones of the plurality of LiDAR sensors at asubstantially same phase angle and to phase lock the second ones of theplurality of LiDAR sensors at a substantially same phase angle.
 6. Thescanning system of claim 1, wherein, at any given time, the first onesof the plurality of LiDAR sensors face in a first direction and thesecond ones of the plurality of LiDAR sensors face a second directionsubstantially opposite the first direction.
 7. The scanning system ofclaim 1, wherein, at any given time, the first and second ones of theplurality of LiDAR sensors face in a substantially same direction. 8.The scanning system of claim 1, wherein the plurality of LiDAR sensorsare positioned at least two corners of the scanning system.
 9. Thescanning system of claim 1, wherein the scanning system comprises avehicle, and wherein the first and second slices respectively correspondto a front side and a rear side of the vehicle.
 10. The scanning systemof claim 1, wherein the scanning system comprises a vehicle, and whereinthe first and second slices respectively correspond to a left side and aright side of the vehicle.
 11. A scanning system comprising: a platform;a plurality of light detection and ranging (LiDAR) sensors mounted onthe platform, each LiDAR sensor of the plurality of LiDAR sensors beingconfigured to generate a point cloud based on interactions of emittedlight with a surrounding environment; a controller configured to controltrigger times of rotations of the plurality of LiDAR sensors, to phaselock first ones of the plurality of LiDAR sensors at a substantiallysame phase angle, and to phase lock second ones of the plurality ofLiDAR sensors at a substantially same phase angle; and a processorconfigured to split a full 360° rotation of each LiDAR sensor into aplurality of slices comprising a first slice and a second slice, and togenerate a coherent fused point cloud by fusing together portions ofpoint clouds of the plurality of LiDAR sensors corresponding to thefirst slice, and fusing together portions of point clouds of theplurality of LiDAR sensors corresponding to the second slice.
 12. Thescanning system of claim 11, wherein the processor is configured togenerate the coherent fused point cloud by: assigning a first timestampto the portions of the point clouds of the plurality of LiDAR sensorscorresponding to the first slice; and assigning a second timestamp tothe portions of the point clouds of the plurality of LiDAR sensorscorresponding to the second slice.
 13. The scanning system of claim 12,wherein separately fusing together portions of the point clouds of theplurality of LiDAR sensors corresponding to the first slice and thesecond slice comprises: fusing together ones of the point cloudsassociated with the first timestamp; and separately fusing together onesof the point clouds associated with the second timestamp.
 14. Thescanning system of claim 11, wherein the first slice of each LiDARsensor is earlier in rotation that the second slice, and wherein theprocessor is configured to process the portions of the point cloudscorresponding to the first slice, while the plurality of LiDAR sensorsare capturing the portions of the point clouds corresponding to thesecond slice.
 15. The scanning system of claim 11, wherein, at any giventime, the first ones of the plurality of LiDAR sensors face in a firstdirection and the second ones of the plurality of LiDAR sensors face asecond direction substantially opposite the first direction.
 16. Thescanning system of claim 11, wherein, at any given time, the first andsecond ones of the plurality of LiDAR sensors face in a substantiallysame direction.
 17. A method of generating a coherent fused point cloudby a scanning system comprising a plurality of light detection andranging (LiDAR) sensors, the method comprising: splitting a full 360°rotation of each LiDAR sensor of the plurality of LiDAR sensors into aplurality of slices comprising a first slice and a second slice, eachLiDAR sensor being configured to generate a point cloud based oninteractions of emitted light with a surrounding environment; phaselocking first ones of the plurality of LiDAR sensors at a substantiallysame phase angle and phase locking second ones of the plurality of LiDARsensors at a substantially same phase angle; and generating a coherentfused point cloud by separately fusing together portions of the pointclouds of the plurality of LiDAR sensors corresponding to the firstslice and the second slice.
 18. The method of claim 17, wherein thefirst and second ones of the plurality of LiDAR sensors face in asubstantially same direction or in substantially opposite directions.